It is well known that nowadays multimedia information is one of the most employed media to communicate information over Internet and other communication and broadcast media. Audio and video content, related with artistic, technical, advertising, musical or personal scope, are shared and published by means of online platforms like Youtube, Vimeo, Spotify, iTunes, Playstore, etc.
Nevertheless, current multimedia player devices are unable to reproduce, in a dynamic and real time fashion, physical and mechanical information related with the action played in the multimedia content when multimedia was recorded or edited. Authors like Herbert Read, call this physical content as “haptic” to refer to the set of physical feelings, neither visual nor acoustic, sensed by individuals, which enrich their sensory experience.
For example, a video file recorded from a formula one car cockpit is not able to reflect any real time information of the physical effects occurring during the recording: bumping, acceleration, breaking, piano passing on curves, etc.
Further, from the point of view of the viewer of a car race, it is not possible to perceive how cars approach and move away from him/her, by feeling real ambience increasing and decreasing vibrations due to engine throttle, as these haptic information is not reproducible using current multimedia players included in a mobile phone or tablet.
On the other hand, wearable equipment, such as helmet or glasses with integrated cameras, is commonly used for recording day life and sport activities like trekking, climbing, biking, diving, skydiving, and many other activities. These recording are oftentimes uploaded to the Internet in order to share them with friends and family or to commercialize them as advertisement, documentaries, etc.
Movement such as jumps, acceleration, deceleration, upwards and downwards movement, rhythm changes and, in general, speed and position changes are easily measurable by mobile devices using integrated sensors and can be stored in the device or sent to a remote server. Other physical parameters that are easily measurable include temperature, pressure and other environmental conditions. All of these physical parameters may be included in a multimedia recording. Unfortunately, despite having several sensors that offer accurate information, current mobile devices and smart phones lack the physical interfaces to reproduce such information in order to be perceived by the user when playing.
Nevertheless, it is well known that physical sensations related with the environment during the recording of a multimedia file can be represented and recognized by the user by means of vibration patterns using a controllable vibrator like the one provided in most current mobile phones and tablets. These techniques are highly spread among game developers although in a limited way (Nintendo Wii or Microsoft Xbox among others) in order for the player to perceive the game environment thus, providing a better game experience when hitting, punching, kicking, explosions, etc.
In the mobile industry there are some applications and companies that allow some level of haptic sensations while playing a game in a smartphone. Some companies, like Immersion Corporation and its TOUCHSENSE® technology, allow editing multimedia content and assigning predefined vibration patterns, stored in libraries, to actions or sequences to previously recorded video or programmed games for mobiles.
One thing that corroborates that haptic information is very important for the user experience is that in a musical theme, for example, the rhythm and the strength of the sound felt in a live concert is one of the most important subjective effects for the user, so rhythm synchronized vibrations proportional to the music level reinforce the sensitivity and the real effect as if user was listening live music. Similarly, when watching sport actions (like kicking the ball in a soccer match, hitting the ball in a tennis court, riding a mountain bike or a horse, running bumps, etc.) a haptic sensation synchronized with the action notably augments the multimedia content getting a much better real sensation for the user.
An important limitation of the current technology is that the current haptic generation systems available require of the human intervention to assign haptic effects to the video. Despite unforeseen technological advancements, the current technology requires that multimedia audio files must be converted through manual conversion. This means that a human designer listens, analyzes, and judges the audio file and selects features of the sampled audio from which to create haptic effects. In fact, Immersion Corporation has numerous related patents (for instance: U.S. Pat. Nos. 8,717,152 and 8,761,915) on semi-automatic haptic generation from audio files using different methods to generate haptic effects. Nevertheless, the method disclosed herein uses only the envelope of the signal and a histogram of the envelope to remove unimportant content based on a user defined threshold, generating a haptic pattern directly proportional to the intensity of the sampled signal envelop. Another difference is that the method here disclosed uses not only audio information to generate haptic content but it can use any other sensor output (acceleration, orientation, video intensity, temperature, pressure, etc) available in the recording device to integrate the physical haptic information with audio and video content synchronously and automatically, in order to be played later or in a streaming way by a haptic multimedia player.
Finally, another possibility is the addition of haptic effects to an interactive reading by providing different ambience effects (storm vibrations, horses steps, etc) for different paragraphs, chapters or scene allowing the enrichment of its content.
Unfortunately, vibration elements integrated in mobile phones, tablets and remote game controllers do not allow control of the vibration strength to more than a few levels making it difficult to provide the user with enough physical information to represent the variety of real sensations. Nevertheless, this method allows a higher than ever variability content using a conventional vibrational element as the ones provided for most mobile devices so providing with the best ever experience in a conventional electronic device.
None of the current applications, patents or devices allow, as this invention does, assigning multilevel and variable physical information to a multimedia file in real time in an automatic way into a multimedia file nor integrate into it to be played lately by mean of another conventional electronic device without making big changes to it or connecting other auxiliary devices.
Thus, based on these basic concepts, the method described in this patent, haptic information related to physical information and its integration in multimedia files can be used in any device or object having at least a vibration element or similar component, controlled by a programmable logic or a computer program, such element being integrated in the device or externally connected to the user by wire or wirelessly.